On-demand Indexing for Large Scientific Data

Enabling search alleviates the need for manual file management, allowing users to find files by the features that are most relevant to them. Despite the need for a comprehensive file system search, most file system indexing work has focused on adapting existing solutions such as the RDBMS or spatial trees to index files. Although a step in the right direction these approaches have focused their testing on typical POSIX metadata which is fully populated, low dimensional, and primarily numeric [2, 3]. By contrast, scientific data is high dimensional, heterogeneous, and very sparse [4]. These findings indicate that approaches such as naive row-major databases, and spatial trees scale poorly to scientific data as they waste space to store null values, have inflexible schemas, and have trouble with many to one values. We propose an indexing technique called On-demand Indexing, which will create relevant indexes at search time if one does not already exist, while simultaneously providing query results. On-demand indexing will achieve scalability by only indexing searched for terms as well as utilizing a storage substrate better suited to high dimensional, heterogeneous, sparse data. Despite the simple approach of indexing everything that is searched for, we believe that like web search [1], file system search term frequency follows a power-law distribution. This means that a majority of searches will be the result of a few frequently searched terms. As a result the system would provide low latency, high precision, high recall search without the storage and computational overhead of indexing all data.